Probabilistic inference serves as a popular model for neural processing. It is still unclear, however, how approximate probabilistic inference can be accurate and scalable to very high-dimensional continuous latent spaces. Especially as typical posteriors for sensory data can be expected to exhibit complex latent dependencies including multiple modes. Here, we study an approach that can efficiently be scaled while maintaining a richly structured posterior approximation under these conditions. As example model we use spike-and-slab sparse coding for V1 processing, and combine latent subspace selection with Gibbs sampling (select-and-sample).

Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties. To address this, we reexamine variational inference from its roots as an optimization problem. We use operators, or functions of functions, to design variational objectives.

The chain structure is also convenient to handle streaming data as we will explain later. We first give a brief introduction to the EP and CEP framework. Step 2. We construct a tilted distribution to combine the true likelihood, Step 3. We project the tilted distribution back to the exponential family, q KL( null p nullq) where q belongs to the exponential family. Step 4. We update the approximation term by's in parallel, and uses damping to avoid divergence. The above computation are very conveniently to implement.

This approach has made variational inference applicable to alarge class ofcomplexgenerativemodels. However,manychallenges remain.

We compare the algorithms on a wide range of large, convolutional and transformer-based neural network architectures.

Bayesian decision theory outlines arigorous framework for making optimal decisions based on maximizing expected utility over a model posterior.

Due to the non-linearity of NNs, no analytic solution to the integral exists, even when the likelihood and the approximate posterior are both Gaussian. A low-cost, unbiased, stochastic approximation can be obtained via Monte Carlo (MC) integration: obtainS samples from the approximate posterior and then compute the empirical expectation of the likelihood w.r.t.